Mainnet client comparison

Delving into client performance.

Table of Contents

• Client distribution
• Client performance
  • Correctness by slot index
• Attestations over time
  • Lighthouse
  • Nimbus
  • Prysm
  • Teku
• Block-packing
  • Myopic redundant aggregates
  • Subset aggregates
• Reward rates since genesis
• Inclusion delay per client

---

This report was compiled with data until epoch 640 (2020-12-04 08:16:23 UTC). We look at the performance of validators who self-declared their client, either writing the client name in their graffiti or with the POAP tag.

All code available here.

Client distribution

Declared client is obtained in the graffiti of produced blocks:

• Either when the graffiti starts with poap and ends with a, b, c, d and e (respectively, Prysm, Lighthouse, Teku, Nimbus and Lodestar).
• Or when the graffiti contains the client name in its string (e.g., teku/v20.11.1).

Since the chain started recently, we do not have a lot of graffitis to scrape from. Additionally, not all graffitis feature the client name or the poap (thanks, Mr. F). This analysis is carried over self-declared clients then.

We have identified the client of 4118 validators, out of 19587 blocks produced.

Client performance

Correctness by slot index

We observe a lot more incorrect head attestations when the attestation is made for the starting slot of a new epoch. We name slot_index the index of the slot in the epoch (from 0 to 31).

Attesters get the head wrong whenever the block they are supposed to attest for is late, and comes much after the attestation was published. We can check which clients are producing these late blocks.

Note that there is a similar issue with targets, since the first block of a new epoch is also likely the candidate target checkpoint.

Since these late blocks seem to happen more often at the start of an epoch than at the end, it is quite clear that epoch processing is at fault, with some clients likely spending more time processing the epoch and unable to publish the block on time.

We can also check over time how the performance of validators on blocks at slot index 0 evolves, again plotting per client who is expected to produce the block at slot index 0.

Attestations over time

In the plots below, we align on the y-axis validators activated at genesis. A point on the plot is coloured in green when the validator has managed to get their attestation included for the epoch given on the x-axis. Otherwise, the point is coloured in red. Note that we do not check for the correctness of the attestation, merely its presence in some block of the beacon chain.

The plots allow us to check when a particular client is experiencing issues, at which point some share of validators of that client will be unable to publish their attestations.

Lighthouse

Nimbus

Prysm

Teku

Block-packing

A block can include at most 128 aggregate attestations. How many aggregate attestations did each client include on average?

Smaller blocks lead to healthier network, as long as they do not leave attestations aside. We check how each client manages redundancy in the next sections.

Myopic redundant aggregates

Myopic redundant aggregates were already published, with the same attesting indices, in a previous block.

Subset aggregates

Subset aggregates are aggregates included in a block which are fully covered by another aggregate included in the same block. Namely, when aggregate 1 has attesting indices \(I\) and aggregate 2 has attesting indices \(J\), aggregate 1 is a subset aggregate when \(I \subset J\).

This analysis is carried until epoch 250 (2020-12-02 14:40:23 UTC).

Lighthouse and Nimbus both score a perfect 0.

Reward rates since genesis

We first look at the reward rates per client since genesis.

Inclusion delay per client

We check the inclusion delay over all validators per client.

We can also check the inclusion delay by block, where instead of looking at first inclusion of the attestation minus the attestation slot, we compute the first inclusion of the attestation minus the earliest block in which this attestation could have been included. Note that the minimum value of the inclusion delay by block is 0.